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ABSTRACT 

When different individuals operating in different 
contexts implement similar innovations, they approach the various 
parts of the innovation in different ways. When assessed according to 
the research procedures used to study innovation implementation at 
the Research and Development Center for Teacher Education at the r ^ 
University of Texas at Austin, the patterns formed by the combinai^tion 
of these modified parts into new, tailored versions of the innovktion 
are called innovation configurations. This report describes the 
five-step process for developing the checklists with which innovation 
configurations can be assessed and evaluated, then delineates some of 
the conceptual and technical problems associated with using the 
process. The five data collecting steps include identifying the 
components of the innovation, verifying the use of these components 
and establishing the range of variations in that use^ refining the 
checklist developed in the first two steps, collecting data, and 
analyzing the data. Discussed here are the following problems 
associated with the procedure: establishing the reliability of the 
checklists, developing suitable data aggregation methods, and 
assessing the degree of change represented by the use of a particular 
version of an innovation. Examples of data types and data 
presentation methods are included. (PGD) 
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A PROCEDURE FOR ASSESSING THE IMPLEMENTATION OF 
INNOVATIONS WITH POSSIBILITIES AND PROBLEMS^'^ 



Susan Heck 

Research and Development Center for Teacher Education 
The University of Texas at Austin 



The purpose of the 1981 AERA symposlunit "Assessing Program Adaptation 
During Implementation: Concepts, Strategies, and Issues," is to present one 
approach to the measurement of Implementation of educational Innovations that 
has been developed at the University of Texas Research and Development Center. 
The approach has evolved out of the Project for Adopting Educational Innova- 
tions' research on change efforts in schools. Initially the Project's research 
focused on the behaviors and attitudes of teachers implementing an innovation. 
It has more recently turned to the innovation itself in an attempt to expand its 
understanding of the change process. The concept of Innovation Configurations' 
grew from the Project's efforts t^^conTeptualize, define and measure innovations 
as they are used by individuals in an organizational setting. 'The history and 
background to the development of Innovation Configurations is contained in Hall 
and Loucks (1981) paper presented at this same symposium, and in other works of 
the Project (Hall ft Loucks, 1978; Heck, et a1., 1981). 



'The research described herein was conducted under contract with the 
National Institute of Education. The opinions expressed are those of the • 
authors and do not necessarily reflect the position or policy of the National 
Institute of Education. No endorsement by the National Institute of Education 
should be inferred. 

^Paper presented at the annual meeting of the American Educational 
Research Association, April 1981, Los Angeles. 
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Innovation Configurations, as we have conceptualized them, represent the 
operational patterns of the Innovation that result from Implementation by dif- 
ferent individuals In different contexts. In the course of early research we 
conducted with different Innovations, we noted that Individuals used parts of 
innovation In different ways. When these parts were put together, a number of 
different patterns emerged, each characterizing a different use of the Innova- 
tion. Ue called these patterns Innovation Configurations.' The means of repre 
senting the parts of the innovation and variations in the use of these parts, 
such that patterns may be derived, is called an Innovation Configuration Check- 
list. ' 

My role in the symposium is to present the procedure that has been devel- 
oped to measure Innovation Configurations^ to delineate some of the problems, 
both conceptual and technical with its use, and to suggest some directions for 
future dialog in the area of Implementation measurement. I discuss first the 
procedure and then the problems. 

I. Procedure for Developing Checklists and Assessing Innovation Configurations 
The procedure for collecting Information from which Innovation Configura- 
tions are therv built, is relatively straightforward. The analysis of the infor 
mation collected, may, or may not, be simple, depending on the purpose of the 
data eollection and the nature of the innovation.- The starting point of the 
procedure is always the consideration and clarification of how the Information 
collected will be used. Information can be used to answer the question," °What 
exactly do adopters of an Innovation do when they use the innovation?" This 
sort of question asks for descriptive information about the behaviors of indivi 
duals as they Implement an Innovation. Such Information is potentially useful 
in a dissemination context when a new program is being explained to nonusers. 



It may also be useful in a staff development context when information is needed 
on what users do (or do not do) before inse.rvice workshops can be planned. 

Another question requiring descriptive information asks, "To what extent 
are innovation adopters using an innovation in a particular way?" The questions 
suggest some norm or standard against which user performance is to be measured. 
This questions may be applied to a staff development context 1n which what a 
user is doing is compared to what the developer of the innovation or a change 
agent working with innovation users intends for the user to do. The question 
suggests a perspective on information-gathering of persons outside the context 
of the innovation user. It is the perspective most commonly aflopted when talk- 

ing about innovations. 

Whether or not the first or second question is asked depends on the pre^- 
scriptiveness of the innovation, as well as situation-specific variables such as 
time-elapsed since implementation, needs of the funding agency, politics of the 
school, etc. By prescriptiveness, we mean the degree to which users are expect- 
ed to use the innovation in pre-specif iced ways with little or no adaptation. 
Users of Innovations that are prescriptive are suppoed to Implement the innova- 
tion faithfully. Implementation is matched to some "ideal" as established by 
someone outside the context of the implementation. The intent, not always 
explicit, of collecting Information about implementation of innovations that are 
prescriptive, is to move adopters toward more fatthful replication of the inno- 
vation. 

Innovations that leave discretion to the individual user or adoption site, 
or even explicitly encourage change, are not prescriptive. It is more difficult 
to evaluate whether or not users are implementing a non -prescriptive innovation . 
according to some preconceived plan. The information needs around a non- 
prescrtptive innovation tend to be purely descriptive, at least initaflly. 



A third order of questions asks, "What are the effects of having imple- 
mented the innovation?" Effects are usually intended as student outcomes. In 
this case, information about implementation is not an end in itself, but a means 
to further analysis using outcome data. Thus, it is important that the informa- 
tion collected about implementation possess the necessary properties to be used 
In multivariate analysis (Taylor & Bianchi, 1981). In short, the type of ques- 
tion asked has bearing on the instrument developed and the analysis performed. ' 
I turn now to the procedure for collecting information about implementation. 

The procedure for collecting information consists of five steps. The flow 
chart on the next page summarizes the general procedure for identifying Innova- 
tion Configurations. (For an extended discussion of the procedure, see Heck, et 
al., "Measuring Innovation Configurations: Procedures and Applications," 
198T). 

STEP 1; Identifying Innovation Components 

The first step requires the identification of components, or parts of the 
innovation. Components are the major operational features of an innovation. 
For instructional innovations, these are usually either materials, teacher be- 
haviors or student activities. 

Component identification is begun by reading descriptive materials about 
the program. Next the developer or program facilitator, or curriculum coordi- 
nator is interviewed. Sometimes developeris are the teachers in the program; 
sometimes they are in the district office. Often they are outside the adoption 
institution, and are in other schools, development agencies, or publishing 
companies. The outcortfe of the first step is a clear picture of what constitutes 
the innovation as key persons want it to be used by teachers or others and a 
tentative list of components and a few variations for each component. 



PROCEDURE FOR IDENTIFYING INNOVATION COMPONENTS. VARIATIONS AND CONFIGURATIONS 
ACTIVITIES . OUTCOMES 



Review written materials describing 
innovation v 



Interview developer for Innovation 
components and variations within 
each component 



Interview and observe a small number 
of users at exemplary site to verify 
developer's component checklist 



Return to developer to reconcile 
points of view presented by ' 
developer and users: establish 
universe of variations; delineate 
"acceptable" and "Unacceptable" 
variations, from developer's viewpoint 
(if desired). 



Interview large number of 
different adopter sites 



"users" In 



General familiarity 
with innovation 



Preliminary checklist 
of Innovation components 
developer's perspective 
with examples of variations; 
Interview questions and 
probes for exemplary site 



Revised checklist with 
questions to ask of 
developer 



Final component checklist^ 



with variations and decision 
points; Interview questions 
and probes for interviewers 
to use In field 




Data for use In developing 
Innovation configurations 



STEP 2; Verification of Components and Variations 

Subsequent to interviewing the developer, users are observed and inter- 
viewed to get a concrete image of what users are doing when they implement and 
interweave the components. Users are probed for the components they believe to 
be essential to the innovation. Different types of users, new, old, at differ* 
ent grade levels, etc., are interviewed so as to elicit as wide a variation in 
implementation as possible. The initial checklist is expanded at this 
point. 

STEP 3: Refinement of Checklist 

The third step in delineating the "tt** that adopters are working wfth is to 
refine the checklist through new discussions with the developer. These serve to 
clarify which are the most important components, to verify variations, to re- 
solve discrepancies between developer and user viewpoints and to standardize 
language and format. At this point, questions and probes to ask of users are 
added. A checklist is now ready for completion by users or for use by persons 
interviewing and/or observing users. A sample checklist Is exhibited on the 
following page. 

STEP 4; Wa Collection 

The data base for use in analysis of components and delineation of Configu- 
rations ma^ be generated from interviews, observations, and/or self-administered 
checklists. Completion of the checklist by the user has the primary advantage 
of making few demands of use^' time and availability. It is best utilized with 
innovations that are simple and require few complex interactions between compo- 
nents. More complex Innovations need the breadth and depth of information that 
an Interview provides. 
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Tutoring Program Checklist 



!• Materials and Equlpmant | 

(t) 1 (2) 

At laast 5 diffarant program | At laast 3 diffarant program 
Mtarlals ara usad with aach . matarlals ara usad with aach 
child aach sasslon* | , child aach sasslon. , 


(S) 

lass than 3 diffarant pro- 
gram matarlals ara usad with 
* aach child aach sasslon* 


2m DIagnoals . 

(1) C2) 

^ ChMdran ara diagnosad . Chlldran ara dfagnoaad 
Individually using a | Individually using taochar 
combination of tasts and . Judgmant only* 
tatichar Judgman^* | 


(3) 

Chlldran ara not diagnosad 

Individually* 4 


3« Racord*Kaap I ng . 

(1) 

Individual Racord Shaat Is . 
usad to Racord diagnosis and p 
prascription* 


(2) 

_ % 

No Individual Racord Shaats 
ara usad* 


4« Usa of Taaching Tachnlquo | 
(f) 1 

Continually raad Justs task | 
according to child naads; 
usas rawards to ralnforca | 
studant succass* . 


(2) 

Doas not continually raad Just 
task according to child 
naads; doas not usa rawards* 

i 


5* Grouping | 

(1) 1 

Childran ara taught In' | 
pairs. 


(2) 

Chlldran ara not taught In 
pairs* 


6« Schaduling | 

(1) 1 C2) 

Childran ara taught for 30 1 Chlldran taught for 30 ml n« 3 
ml nut as 3 tlmas par waak. . timas par waak, tima for aach 
Each sasslon Is aqually | chUd and aach task varlas 
4Jvldad batwaan chlldran. . slightly whan nacassary* 


(3) 

Chlldran not taught for 30 
• min* par waak 3 tlmas par 
waak, or tIma for aach child 
and^aach task varlas aark* 
adiy or Is net consldarad* 



CODE: Variations to tha right ara unaccaptabia; variations to tha laft ara acdaptabia* 

Variations ta tha laft ara Idaal* as pffiascrlbad by tha davalopar* . 



Interviewing allows the Individual to define the Innovation as she/he sees 
It, In terms of her/his relationship with this Innovation, without the restric- 
tion of component categories Imposed from the outside. In an Interview, the 
checklist becomes both a guide for the Interview and a tool for recording the 
Information elicited from the user. 

Observation has particular value when an Innovation Involves multiple user 
roles, or has components that call for Interactive processes. In combination 
with Interviewing or self -completion. It allows for a broader perspective of the 
Innovation and a sense of the context which can be helpful In Interpreting user 
responses. Observation also provides a means of validating the Information 
collected by interview or pen and paper methods. 

STEP 5: Data Analysis 

There are many ways to analyze the Information collected on a component 
checklist, though the most common type of analysis Is the simple computation of 
component frequencies. Profiles of how components are used by teachers within a 
team, grade level, school or district are made frOm the raw tallies. The 
examples on the following pages show data summarized within a single school for 
11 teachers and across 11 schools for 92 teachers. The Innovation being Imple- 
mented was an elementary science curriculum. Each of the components of the 
Innovation ^as operational Ized grating points for each variation were estab- 
lished by principals and district coordinators after extensive observation and 
Interviews with teachers. Data was collected by Interview and observation. 
Teacher data was aggregated within schools, and data from individual schools was 
aggregated across the district. The final data was used both for reporting to 
the Board and for planning inservice. 

" ' 11 



Sample Building Suwmyy Sheet 
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WELL 
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GOOD 
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2 
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P3 



1. Tine is devoted to^' 6clenjp;e 




* 


** 


* 


< ** 


— ■ . "ig), i T 

2. Science is taught according to 
R-1 Guide 


*** 
*** 


*** 
** 








3. Assessment of pupil learning 




*** 
^* 








4. Integration of basic skills 





***** 
**** 


* 






5« ^he outdoor classroom jl^useS as 
recozaended , ' 




. *** 


*** 

V 

'* 


** 

() 




6. Recorsnended materials, equipment 
and media are avail^able 






*** 
** 


*** 
* 


*^ 


7^ Inservicing and financial 
arrangements have been made 




* 


ik** 
** 


*** 
** 




8.^ Long and short range planning 




*** 


*** 
*** 


** 


5 


9. Use of class time ^ 


** 


** 


**** 


** 


* 


10. TeachSr-Pupil interaction 
facilitates program 


*** 


**** 


**** 






■1 

11. Classroom environment facili- 
tates program 




*** 


*** 


*** 


** 


12. Instruction is sequenced tb 
facilitate the guided inquiry 
learjning approach 


** 


**** 
* 


**** 







School 



Winter Elementary 



Teacher 



all 3t A» 5» 6 teachers 



R-1 Program Evaluation ^ Jp 

Revised Elementary Science Program Evaluation .1/80 



The Average Extent of Implementation on Each Component for All Classrooms 



Outside 0»tting W«ll Best 

Intended A Good " On Tlie Practices 

Program Start Way Working 
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!• Tlxnt Is dttvottd to 
sclenct 
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2. Scitnce is taught 
according to &»1 
Guide 






- 














3e Assessment of pupil 
learning : • 










- 










4e Integration of 
oasic SKxUs 
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5. Tha outdoor class- 
room is used as 
recommended 










» 










6. Recommended materi- 
als, equipment a!nd 
media are available 






















7. Inservicing and 
financial arrange- 
ments have been 
made 








« 














8. Long and short- 






















9. Use of class time 








, i 






10. Teacher-Pupil 
interaction 
facilitates nroeram 
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He Classroom environ- 
ment facilitates 










• 
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^ ^ Jt^ 




, 12. Instruction is 














sequenced to 
facilitate the 
guided inquiry 
learning anoroach 
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K-l Program Evaluation 
■ Elenentary Science Program Etratuation l/ftQ 



A further analysis step involves the development of Innovation Configura- 
tions either by hand analysis of the checklist data or by the use of computer 
pattern analysis techniques. Whetheri^r not the development of Innov^on Con- 
figurations ts an appropriate next step, again, depends on one's information . 
needs. As a heuristic. Configurations can be useful in describing how groupsof 
users approach implementaiion. It can simplify and consolidate a great deal of 
detailed information, for example, for reporting purposes to a school board or 
outside funding agency. While a* useful summary measure, it cannot be used as^a 
diagnostic for individual teachers because the groupings mask individual differ- 
enceis. Data at the individual level must be used for specifics on each teach- 
er. . " , 

<■ Innovation Configurations can also be used to relate implementation pat- 
tern^ to o*tj:omes. This hijs been done with the Configurations developed from a 
Skills Achieveqient Monitoring system (George & Hord, 1980) that was develope^ 
^or the purpose bf assessing the mathematics skills of students. Three classes 
of innovation users were developed clinically by the interviewers who had col- 
lected the information on the checklist. The component, "Use of Objectives* was 
the starting point for discrimination and initial placement of users. The three 
basic groupings, called "high," "medium," and "low" implementation, were entered 
in the computer and a non-hierarchical cluster analysis run. A vector of aver- 
ages on each item was formed for each of the three groups. Percentage of users 
employing a particular practice was computed for each component within each of 

the-tbree 5r<)up4ngs. ,Jhe . results of. -the..clij[Ster.tog_ pr^^^^ 

pages 11 and 12. 

An analysis of covariance .was performed using the final achievement test of 
^he school year as the dependent variable. The three implementation groups 
served as the independent variable; the initial achievement test, grade level 

11 
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^Results of Hierarchical Cluster Analysis 
on" SAM Configuration Checklist 



Objectives , ^. 

1 Uses the SAM objectives as the primary curriculum guide for 
ongoing instruction. 

2 Devotes class time to teaching some or all SAM objectives 
for ongoing math instruction -- not remediation. 

3 Increases attention to teaching SAM objectives immediately 
previous to Sm testing (i.e.. "preps" for SAM tests). 

4 Teaches math objectives other than SAM objectives (whether 
or not SAM o|||ectives are taught). 

5 Instructs resource room students in SAM objectives. 



Instructional Materials 

1 Uses pre-packaged materials (IMP or similar) keyed to SAM 
objectives for ongoi ng math instruction. 

2 Uses personally grouped materials keyed to SAM objectives 
for ongoing math instruction. 



Testing 

1 Administers tests specifically focused on- SAM objectives be- 
tween SAM tests. 

2 Administers other math tests besides SAM tests. 

3 Administers more than one level of the SAM to individual 
students (i.e., a student takes two SAM tests simultaneously) 

4 Moves students from one level of the SAM to another during 
the year. 



Printouts 

1 Sees that students each receive a copy of each SAM printout. 

■ • ' 

2 Provides that each student has a readily accessible record of 
performance (printout or chart) on previous SAM tests. 
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3 Expects students to take each (every) printout to parents. 

4 Sends printouts home with students at end of year (regard- 
less of other times). - 

5 Requests parents to sign to show they have received printouts. ^ 

6 Shares SAM printouts with child's other teachers (e.a.. Title 
I). ^ . 

* 

7 Posts SAM printouts in classroom (e.g., on the wall). 



Remediation 

1 Requires students to show mastery or work toward mastery of 
objectives missed on SMA tests. 

2 Creates problems or exercises "on the spot" to reteach missed 
objectives. 

3 Uses pre-packaged materials keys to SAM objectives (IMP or 
similar)! for remediation work. 

4 Uses personally grouped materials kleyed to SAM objectives 
for remediation. , . ^ 

5 Reviews SAM results with class as a whole within a few days 
of their return. 

6 Reviews SAM results with individual students within a few 
days of their return. 

7 Focuses remediation on whole class (based on SAM results). 

^ ■ ■ < . 

8 Forms small groups based on SAM results for the purpose of 
remediation. 

9 Focuses remediation on individual students (based on SAM 
results). 

10 Cpmpares previous results on test with current results. 

11 Assists students to be aware of progress made since last test 
(or over the year). 
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and sex as the covarlates. It was shown that the ^medlum*^ Implementation group 
correlated highest with achievement scores. The results of this analysis were 
used In application for validation of the Innovation at the state and federat 
level. They were further used by developers of the 1nnovatix)n and tieclslon 
makers to see which practices had gained widespread use and which seemed to be 
most related to outcomes. This Information was communicated to new schools who 
were c6as1der1ngi|^dopt1on of the Innovation. 

A second example where outcomes were related to Innovation^ Configurations 
was in an experimental Bilingual program (Butler, 1980). The purpose of the 
program wa$ to Implement three Instructional models, one each In two of the six 
participating schools. The Instructional models were defined' a priori and 
information from a checklist was used to classify users as belonging to one of 
the five models. An analysis of covarlance was performed that^ while producing 
relatively few results that could be educationally significant, did reveal gains 
of the Project groups that fairly consistently outweighed those of the comparl* 
son group. The data supported the recommendation that the District consider 
utilization of the cheaper of the models, since neither seemed related to more 
significant gains. 

While both these examples used Configurations as a measure of Implementa- 
tion that could be related to outcomes, they were quite different In their 
approaches. One, the Student Achievement Monitoring system, derived both com- 
ponents and Configurations from the interviews with teachers; the other, the 
Biliiigual program, used theoretical TOdels to-establish thre Confi 
fit the data to these models. Both examples were useful, yet also generated 
problems, which are discussed In the next section. 
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II. Som^' Probleffis Inherent in th6 Prooedurej Caveats and Posslbnities 

A. Checklist Reliabilities . 

One of the questions we have asked ourselves concerning the Implementation 
assessment procedure presented above. Is how rel^lable It Is to assess Imp'le- 
mentation at a particular point In time. In seviiral Instances we have tried to 
ascertain the answer to this question, some in mdr^e formal ways than others, 
^n the first Instance, the Innovation was the Skills Achievement Monitoring 
. system cnpntiwed previously (George & Hord, 1980). Jhe monitor ing system as 
finally evolved, consisted of criterion-referenced tests, a computer program for 
scoring the tests, and printout resultst The Innovation was non-prescriptive in 
that' it left up to teachers what they would do with the tests and test results. 
The checklist that was used with teachers served as a guide for the interview, 
but the final components could be derived only after receiving information from 
teachers as to what the innovation was. Interview tapes were analyzed and a ^ 
29-item checklist derived from the teachers' statements about their use of the 
innovation. As the interview had been extensive and contained the same cate- 
. gories of behavior as the 29-item checklist, it was decided to try to make 
inferences on those items not specifically addressed in the Interview. Most 
teachers gave very specific descriptions of their practices in the classroom, 
thus making the completion of the final checklist from inferences a possibility. 
Six interviews were rated independently by another person to assess the reli- 

ability of the ratings. <)fv these tapes, there was an^ 8^^^^ 

False rating for items on which one or both raters Indicated an Inference was 
nec ess Ary. Ninety-four percent agreement was found on items for which both 
raters agreed no inference was necessary. 

> On the one hand, these statistics suggest that good reliabilities can be 

roaintalned even under "high inference** conditions. On the other, it should be 

O noted that the items on the checklist were all dichotomous. The user was rated 

ERIC - ,5 18 



•s either "doing" or "not doing" an Item. Consequently, the rater did not have 
to chdose between different variations, therein making ratings easier and rell- 
abilities more likely. We do not know what the retlabllltles might be with more 
variations for each component. 

A second setting in which we looked at checklist reliabilities was a 
nationwide study of dissemination strategies of four federally funded programs- 
(Crandallt D.P., et aT., 1981). A subset of the larger sample was used for an 
ethnographic study. Field workers spent up to a week three times a year at a 
site both interviewing and observing people involved in the implementation of 
the innovation under study. Some of the teachers at the ethnographic sites had 
also participated in the larger study in which an interviewer using the compo- 
nent checklist had asked teachers about their use of the innovation. Ethno- 
graphic site workers filled out the same checklist for the teachers that had 
beeri interviewed earlier by other field workers. 

The few matching checklists that we do have contain discrepancies in the 
ratings of the ethnographic and interview field workers. This maty be due to the 
fact that the field worker gathered in^or^mation to complete the checklist at a 
different time from when the interviewer was present. It is more likely, in our 
opinion, that the two data collectors imputed different meaning to the component 
and their variations, particularly where the , component described teaching 
processes and student Interactions. Ethnographic field workers, by definition, 
J^n^wJ^^r at the site, and the teachers using 

the innovation than the interviewers. They were more likely to interpret the 
component from the user's point of view to search if a teacher ijrere using the 
spirit, if not the letter of a component. The possibility of unreliable inform 
mation, in the sense of differing information, presented by two different 
sources goes back to the question of whose perspective does one capture on the 
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checklist? Does one want to incorporate- the meaning of components for indivi- 
dual users in their daily routines, and if so, how? 

' The last Instance in which reliabilities were attempted was a different 
case. The innovation in point was the bilingual orogram mentioned earlier 
(Butler, 1980). The purpose of the program was to implement 3 instructional 
models, on each in two of the six participating ^chools. The checklist con- 
tained five components. The variations within each component were defined as 
belonging to one of the three instructional models. Decision rules concerning 
which combination of variations fell into which Configurations, or instructional 
model, were established. Interviewers recorded their overall judgement of a 
teacher's position on the continuum of models which was later expanded to con- 
tain two mixed models, as well as to classify each teacher on the basis of each 
component rated. Each teacher interviewed was also observed and rated on the 
same checklist by district resource staff the same week as the implementation 
interview. Reliabilities on the overall rating on the instructional model 
implemented by the-teacher was over 90X. Reliabilities were not computed on 
individual component ratings. 

This case is different from the others in that reliabilities were estab- 
lished on overall configurations rather than on individual components. This 
approach was possible because the instructional' models or Configurations had 
intentionally been built into the program from the beginning. 

Where the question of interest is whether or not a general model or config- 
uration is being implemented, then it is likely that the checklist is a reliable 
instrument. When information is needed about specific components or behaviors, 
it is more likely that there will be disagreement between raters, due in part to 
lack of specificity of the desired behavior and the resulting difference in 
meaning that different interviewers and observers impute to the component varia- 
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tion. There may be no simple answer to this problem, for the more one moves 
.toward precision in defining behaviors, the more likely one is to loose the„ 
Interactfon of that behavior of the rest of the behaviors, setting, and context 
In which it is embedded. 

B. Data Aggregation ' 

Another area in which we have raised questions Is in the aggregation of 
data across individuals. At times we want to know about overall implementation 
at a site, and can summarize component data across users for staff development 
or formative evaluation purposes, One simply indicates the number or percentage 
of users implementing a component 'tn a given way and displays the information in 
matrix form. 

The problem is somewhat more complex vShen trying to relate the innovation 
to student outcor^es and/or^sumnilr izlng data acrflsg^^ites. jand (even innovatljons. 
In the case of the bilingual education project described above, the procedure 
for relating Innovation Configurations to student outcomes via an analysis of 
covariance was relatively straightforward. With only six components on the 
checklist, clear cut decision rules for assignment of behaviors to Configura- 
tions, it was possible to relate program implementation information to student 
outcomes on standardized achievement tes!ts. When Configurations are derived 
subsequent to data collection, however, through some sort of cluster analysis. 

Is 

as was the case of the Student Achievement Monitoring system described above, 
the concern with the stability of groups and the possibility of reproducing the 
same result with different groupings arise^. More work on the, formation of^' 
Configurations through cluster techniques is currently underwa^y at the CenteK. 

In the national study of innovation dissemination mentioned above \ 
(Crandall, D.P., et al., 1981), another form of aggregation problem arose 

» * 
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(Taylor fc Bidnchi, 1981). One objective of the study was to compare extent of 
ImplementatloiMneasures across schools. Innovations and federal dissemination 
strategies In terms of length of adoption, type of Innovation, extent of the 
adoption locally, level of local school efforts, chaAracterlstlcs of teacher 
participation, administrative support, demographic factors of schools and char- 
acteristics of states. Checklists were designed for 63 different Innovations. 
Each checklist measured fidelity to some Ideal Implementation. Variations for 
each component were expressed on a continuum with each variation moving farther 
away from the ideal. A score for individuals was arrived at by summing the 
scores of that individual on each component. Site scores represented an average 
extent of implementation for all individuals at that site. Besides questions 
concerning the appropriateness of the site score per se, it was found that the 
loss of information Incurred when aggregating individual scores to the site 
level and using site scores in a multivariate anj^lysis washed out those effects 
that were later found in data left at the individual level. The problems of 
missing data, the trade off between lost cases and dropped variables, multiple 
sources of sampling error are not new to the checklist. We point them out, how- 
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ever, simply to Illustrate the need for caution in using the Instrument in a 
research/evaluation context without careful consideration of the problems that 
are likely to arise. 

! "'' ■ , ^ ■'■ • ' 

C._JHTg^elationsh1p of Past and Present Practice; The Area of "Change" 

A third area which has been problematic in the analysis and Interpretation 
of Information on the checklist is the area of "change." An obvious point, but 
one that can sometimes be lost, is that the information derived from interview- 
ing and observing teachers characterizes what they are doing at present with the 
innovation. The information does not Indicate what the user was doing prior to 



the adoption of the innovation. It cannot convey whether or riot present prac* 
tice Is different from past practice, although certainly In the course of an 



doing before implementing the innovation. In a change effort where fidelity to 
some new program is desired. At is Important to consider where teachers started 
as well as Where they are currently in the implementation effort. Expectations 
for what can realistically be achieved by a teacher (or is desirable), and the 
nature of technical assistance, depend largely on users* starting points. The 
checklist can be used to structure questionis about what teachers were ^doing 
previously, but it is important to remerriber that the components relate to the 
present program. 

A second point concerning change relates to what the student receives in a 
new program. Does change exist for the student? Take the case of a counselor 
turned to stud^1^> tutor in a remedial reading program for disadvantaged chil- 



cant. For the student, who has been receiving some form of remedial reading 
over the past 3 years, the change ma^y not be significant. Do we expect that the 
teacher will need some support in the role change? Probably yes. Do we expect 
that the student will perform better under the new reading program? Not neces* 
sarily if the program is not •'better" in some way for the student . Once again, 
we recommend caution in using information from the checklist in talking about 
"change." The information does not necessarily imply change; arid change. If it 
does exist, does not necessarily bring improvement. The area of change and what 
it means to both users and students relates to our fourth area of concern, the 
re|evance of the innovation, defined as a set of behaviors, for outcomes. 

A final area in which we have given some thought to the uses and limits of 
the procedures described above concerns the relationship of the information on 



Interview with a teacher, a teacher will share information about what she/he was 




change is probably signifi* 
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the checklist to the total learner environment In the classroom and school. The 
comi^onents of an Innovation represij^nt the behavioral aspects of the Innovation, 
purposely designed to be distinct from the philosophy, implementation require- 
ments and perceived attributes of the innovation. Surely t however, these other 
categories and approaches for describing Innovative programs^ procedures, and 
practices are also Important for the Implementation process. The philosophical 
orientation, the values and beliefs associated with the innovation, and their 
congruence with those of the adopters are important for the resistance to and 
acceptance of the Innovation. Indeed, persons such as Julian (1980) and 
Leithwood (1980) consider the philosophy of the Innovation Integral to the Inno- 
vation. Implementation Includes acceptance and utilization of that philosophy. 
Implementation requirements, the training, materials and facilities needed to 
support the Implementation effort are considered by some to be part of the Inno- 
vation. In their absence, the Implementation effort cannot occur. Finally, how 
users perceive an Innovation, Its utility, advantage, and compatibility with 
their own roles and agenda ^Syjmportant. From a dissemination point of view, 
the perceived attributes of an Innovation are Important for teachers' willing- 
ness to Implement the Innovation, or for the decision to decide not to. A com- 
ponent checklist purposely does not Include these aspects or definition of an 
Innovation. It focuses on behavior of, users. Although we believe user be- 
haviors are essential to the successful' Implementation of Innovations, we recog- 
nize the Importance of other viewpoints and definition In trying to understand 
and Implement Innovations. 

Further, it Is Important to remind ourselves, that teaching and learning 
include a variety of instr^jctional' dimensions that may or maiy not overlap with 
the innovation and which may be more important to student outcomes than the 
innovation itself (Leinhardt, 1980). For innovations that encompass a total 
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cfassrocxn or school such as an alternative high school or a kindergart^n/first 
grade program to prevent school failure or a bilingual program, the point may 
not be of much Importance because the program Incorporates the Instructional 
dimensions to which children are exposed. But many Innovations are not of such 
all-encompassing magnltji^ilf^ ^ They touch upon one of several subject areas to 
Mhlch a student Is expos|ed[ or one of the many management strategies used by a 
teacher, or one type, of stU^dent -teacher Interaction. Far more transpires In the 
course of the whole day ft>r both student and teacher than the innovation Itself. 
In such cases It ma^y be Inappropriate, to expect outcomes associated wit^~the 
Innovation. Ue need to look at the whole learning context of the classroom and 
look for those factors In addition to the Innovation that Influence how and what 
students learn. Innovation Configurations Is both a metaphor and a tool that Is 
useful In defining, and measuring Innovations, but It must be- used In concert 
with other concepts and measures to get at the complex nature of change. 
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